Distributed Construction through Participatory Design

We present the implementation of Distributed Constructionism (DC) through a Participatory Design (PD) methodology for an Online Learning community. The students collaborate on the content and functionality development of an online Modern Greek language course, peer review and publish content contributions, and participate in participatory design teams. Participatory design was implemented as a four step process (a) Build bridges with the intended users, (b) Map user needs and suggestions to the system, (c) Develop a prototype, (d) Integrate feedback and continue the iteration. Distributed Constructionism was implemented to enhance the learning experience and community development. Finally Social Network Analysis is employed to quantitatively measure the strength of the online community established and the key role our participatory design team participants played in sustaining this online community. Distributed Constructionism through Participatory Design 165 Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. INTRODUCTION The contribution of this chapter is the presentation of an empirical study of an online learning community collaborating with the design team of the course under the Participatory Design methodology. We measure the student participation during the different phases of the design process, and the changes in their behavior when new design elements are introduced. We implement the different phases of Participatory Design methodology using a four-stage process: (a) Build bridges with the intended users, (b) Map user needs and suggestions to the system, (c) Develop a prototype, (d) Integrate feedback and continue the iteration. Our participatory design methodology was carried out by following the Distributed Constructionism pedagogical theory, and took advantage of the online and distributed nature of the student community to asynchronously design, implement and study the course. Finally we use Social Network Analysis (SNA) to evaluate the student participation on the online discussion forum. We discover that the students who volunteer to contribute in the participatory design teams and contribute most of the additional material are also the central nodes of the SNA analysis. We conclude that the online discussion board can promote student collaboration and the identification of the key community users who can participate productively in Participation Design activities. Participatory Design Participatory design (PD) (often termed the “Scandinavian Challenge”) (Bjerknes et al., 1987) refers to a design approach that focuses on the intended user of the service or product, and advocates the active involvement of users throughout the design process. User involvement is seen as critical both because users are the experts in the work practices supported by these technologies and because users ultimately will be the ones creating new practices in response to new technologies (Ellis et al., 1998). Blomberg and Henderson (1990) characterize the PD approach as advocating three tenets: • The goal is to improve the quality of life. • The orientation is collaborative. • The process is iterative. Distributed Constructionism Distributed Constructionism (Resnick, 1996) extends the Constructionism theory (Papert, 1991, 1993) to knowledge building communities, where the online learning community (instead of one student) collaboratively constructs knowledge artifacts (Resnick, 1996). The three major activities of DC, within the context of an online learning community are (Resnick, 1996): 1. Discussing Constructions: Students discuss their constructions during the design, implementation, evaluation and reiteration phases 2. Sharing Constructions: Web-based systems allow students to share their constructions and make them part of the shared knowledge. 3. Collaborating on Constructions: The community can use online communication to collaborate on the design and development of the knowledge artifacts. 166 Zaphiris, Zacharia and Rajasekaran Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. The iterative structure of our Participatory Design approach enhanced Distributed Constructionism among the users of the system. The knowledge artifacts contributed to the course, enhanced both the learning experience of the users and the content and functionality of the course itself. This chapter is structured around four main sections. First, our PD design approach is presented and its linkage to the Distributed Constructionism pedagogical theory specified, then the results of the Social Network Analysis of our course are presented. In the third section the results of a user-centered evaluation of the course are also put forward and the chapter ends with a set of conclusions and suggestions for directions for future research. DESIGN APPROACH Our focus has been to design an online learning community around a Computer Aided Language Learning (CALL) course. We believe that online interaction and community would increase users’ motivation, commitment and satisfaction with the online course. The Participatory Design methodology blends nicely with our goal. In particular, involving users during system development is thought to lead to greater user commitment, acceptance, usage, and satisfaction with the system (Baroudi et al., 1986). In the design phase of the online course, we implemented PD as a four-step process (Ellis et al., 1998). 1. Building bridges with the intended users: This step opened lines of communication between intended users and the development team. Specifically, this step involved the initialization of a multidisciplinary development team, identifying key groups of end users, and creating new methods of communication with users. The development team in this project came out of the Kypros-Net (Kypros-Net Inc., 2002) group. Through our involvement in Cyprus and Greece related projects, they had longstanding relations with the intended user community. The intended users have been especially people of the Greek Diaspora, travelers to Cyprus and Greece and other Greek speaking areas and people who are generally interested in the Greek culture and language or languages in general. In our case, bridges with the intended users were build through our years of work at providing information about Cyprus through the web pages of Kypros-Net who primarily attracts the same user population as our intended Greek language online course. 2. Mapping user needs and suggestions to the system: Our conceptual design model has been “to design an effective online Greek language course that can build and sustain an online learning community of students.” Based on the questions and inquiries we received from our users we tried to match their needs (they wanted an easy to follow, both elementary and advanced course that they could attend at their own pace) with our conceptual design model. 3. Developing a prototype: The project consists of 105 audio files, which were originally recorded as radio lessons in Modern Greek for English speakers in the 1960s. The lessons were retrieved from the archives of the Cyprus Broadcasting Corporation, digitized in Distributed Constructionism through Participatory Design 167 Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Real Audio 5.0 format and published online through the course. Although an optional textbook accompanied the original radio lessons, the online lessons were designed as a complete stand-alone course. We used several tools to assist students with the lessons, including an online English-Greek-English dictionary, a Greek spell checker and a web-based discussion board. The discussion boards served as the foundation for creating a community of online students and enhanced the learning experience with Distributed Constructionism. 4. Integrating feedback and continuing the cycle: Feedback from our users and suggestions are continuously incorporated into our design through a series of additions and corrections. For example, we were asked to add an online notes section and to encode some files again because they were corrupted. An important element in the participatory design methodology is the direct involvement of the user in all stages of the design process. We kept the users involved by participating in the discussion boards, and sharing with them design and development plans for the course. COURSE EVOLUTION WITH DISTRIBUTED CONSTRUCTIONISM The students of the audio courses included people with no knowledge of Greek language, bilingual members of the Greek Diaspora, as well as high-school professors of non-Greek language. These students created an open online community whose collaboration has boosted the learning experience of the whole community. The web-based discussion board has proven to be the most constructive tool for the students’ learning experience and the main source of feedback for the maintainers of the project. The experiences shared on the discussion board included tricks and tips on how to record the audio files, installation of Greek fonts, learning methodologies and questions about the Greek language itself that arise from the lessons. The experienced users (some of them were retired teachers of foreign languages) had taken a lead role in the vast majority of the threads on the discussion board, answering most of the questions and encouraging the beginners to study the lessons further. They have also become the communication interface between the maintainers of the project and the community’s requests. At some point, the users started exchanging, through email, written notes taken by the experienced users. They also used the discussion board to announce the availability of their personal notes. This behavior suggests that we must provide (and we did) the users with the capability to post their notes on the project’s site. The students had initiated Distributed Constructionism themselves. The course designers only provided technical support to facilitate the students’ construction activities. Discussing the Constructions The course designers offered to provide publishing access to the online course to whoever wanted to contribute their material. Five users asked to be given access. As we will see later in the Social Network Analysis of the community, these users also had 168 Zaphiris, Zacharia and Rajasekaran Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. a central role in the discussion board. Consequently, the five users with the two course designers constituted the Participatory Design team. The PD team solicited contributions from the user community. The users suggested that they should transcribe the audio lessons, and compile verb lists, vocabulary lists, and grammatical notes for each lesson. Sharing the Constructions All the user contributions were shared in the common area of the online course. The users members of the PD team regularly posted notices on the discussion board about new material for the course. Also, other less active users chose to offer contributions for the course, by posting on the discussion board, rather than contacting the PD team. Collaborating on the Constructions The user members of the PD team did not include any native speakers of Greek. They were all learning the language through the online course, and at that stage, they were primarily dependent on the audio lessons. In order to ensure the quality of the new material before publishing them on the course web site, the user members of the PD team implemented a peer review process. A group of seven users, that included the five central user PD team members, reviewed and corrected all the material before posting them on the web site. Each of the seven users offered to transcribe a number of the 105 Audio lessons, and two of them also offered to provide verb and vocabulary lists. However, all materials were posted in a private area first, reviewed by the seven user members of the PD team, and posted on the web site when the five PD users were satisfied with the quality. Then the two PD course designers, who were both native Greek speakers, would go over the already published material, and make sure that it is correct. Most of the mistakes we had to correct were spelling mistakes, and we rarely had to correct grammatical mistakes. Two months after the Distributed Constructionism effort started, students of the audio lessons managed to transcribe 81 out of the 105 lessons, correct them through the peer review process among themselves and post them on the project’s web site. Six months later, the students had transcribed and peer reviewed all 105 lessons. The knowledge constructed attracted significant user attention. As we can see from the accesses to the audio lessons, the language tools, and the total access of the message board and the notes pages all kept increasing exponentially. However, once we allowed our users to publish their own notes, there was a dramatic shift of traffic from the message board to the notes pages. In our view this is due to the fact that the users did not need any more to visit the discussion board to find out where other users had posted their notes. All the content was already aggregated and organized in a central location. The course’s popularity is apparent from the fact that the course currently has over 20,000 registered students who actively participate in an online community, which evolved around the course. SOCIAL NETWORK ANALYSIS Social network analysis (SNA) is the study of social relationships between a set of actors (Baroudi et al., 1986). Dekker (2002) enumerates the following four goals of SNA: Distributed Constructionism through Participatory Design 169 Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. • To visualize communication and relationships between people and/or groups through diagrams. • To study the factors that influence relationships and how they correlate relationships. • To derive implications of data, like bottlenecks in flow of information. • Finally, to make recommendations to improve communication and workflow in an organization. SNA is backed by social sciences and strong mathematical theories like graph theory and matrix algebra (Borgatti, 1999), which makes it applicable to analytical approaches and empirical methods. SNA uses two approaches, namely, ego-centered analysis and whole-network analysis. In ego-centered analysis, the focus is on the individual rather than on the whole network and normally involves only a random sample of network population. Whereas, in whole-network analysis, a survey of full population of network is done, and, hence, facilitates conceptualization of complete network. In order to test the hypothesis, which involves all members of the online community, the whole-network approach was employed. SNA uses various concepts to evaluate different network properties (Baroudi et al., 1986), such as: centrality, connectivity and cliques, each of which pertain to particular dimension of the network. This study has employed a few of the important concepts in testing the hypothesis and to reveal any useful implications over the system performance. Figure 1: Access Statistics for Greek-Online Site 170 Zaphiris, Zacharia and Rajasekaran Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Degree Centrality This assesses the power of an actor (in the online course evaluation an actor is a student) based on the number of alters (other actors) that actor is directly connected to. If an actor receives many ties (relationships), i.e., has a high in-degree, then he is regarded to have “high-prestige” in the network (e.g., Figure 2). Similarly, when an actor makes contact with many alters, i.e., displays high out-degree, then he is regarded to be “highly influential” (e.g., Figure 3). Cliques A clique is a subset of actors in a network who are more closely tied to each other than to the other actors who are not part of the subset. Many techniques in SNA involve complex and voluminous mathematical calculations. This study has used a GUI based network analysis tool called Netminer for Windows (Netminer, 2002). This tool not only handles large volumes of data and performs complex calculations but also helps in the visualization of the social network structures and positions of actors within the network. The visualization of network is mainly achieved using Sociograms, which are graphs with nodes and lines for actors and ties respectively. The tool also provides numerical data analysis in the form of reports. SOCIAL NETWORK ANALYSIS RATIONALE People-intensive social systems like Web-Based Training Systems (WBT) are open-natured and behave unpredictably with their environment. In other words, these systems are less responsive to well defined scientific methods. Hence a shift in focus from technical to nontechnical approaches like social network analysis (SNA), which is principally used in communication science, (Baroudi et al., 1986) is necessary. SNA offers an easy analytical method to monitor and analyze group dynamics, whose results can be Figure 2: In-Degree Centrality Measure Distributed Constructionism through Participatory Design 171 Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. translated into implications of social network over the performance and development of the system in consideration (Baroudi et al., 1986; Garton et al., 1999). Following this, an attempt has been made to evaluate the applicability of SNA as an analytical and empirical method to analyse the interactions within an online community developed through computer mediated communications. This is done by studying the postings made over the discussion board associated with our WBT System by the registered members of the system. Social network analysis helps in identifying useful interaction patterns and relationships among the online community members, whose results are employed to reverse-engineer subsequent system designing and development process. Furthermore, using SNA techniques, the following hypothesis, which asserts the importance of Participatory Design (PD) methodology, is also tested: By involving users actively during the design and development phases of a system, a sense of ownership can be cultivated within the users, which consequently will influence them to play key roles in the system. For example, by taking a lead role in building an online community or by using the system extensively. By testing the above hypothesis one can ensure that up-fronting time and effort in Participatory Design (PD) can be significantly rewarding and consequently motivate system owners to implement PD in the ongoing process of system improvement and maintenance. Results/Analysis of SNA For the purpose of SNA, a selection of postings (the collection of initial interactions between the online community members) was subjected to analysis. These postings were carefully examined, categorized and tabulated with details of type of posting made, target Figure 3: Co-Membership Clique Analysis 172 Zaphiris, Zacharia and Rajasekaran Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. of the posting, etc. From the table of postings, information is then transformed in the form of a data set, usable by Netminer for performing SNA. We masked the users’ names and email addresses with dummy identities to preserve privacy. In particular, those members who have actively contributed in the participatory design (represented with dummy variables U1, U7, U17, U19, U21, U27, U36 in the analysis that follows) are observed throughout the analysis so as to test the hypothesis. The community analyzed is comprised of 43 members, out of which five are participatory design team members and two are the course designers. A rigorous analysis has revealed much useful information from which constructive inferences and assertions can be made. An in-degree visualization of the community at scale three (Figure 2) groups three of the five PD team users at the centre, depicting that they have the highest indegree among seven others, followed by other two at the next level. Similarly, the outdegree centrality analysis at scale three also groups the three PD team members with the others, as having high out-degree. Both of these results clearly depict that the majority of PD team users had large contributions in the online community interactions. The subgroup analysis to identify the “cliques” present in the network shows () a group of five as those who are present in all of the 35 cliques formed in the network. This can be verified from the cliques table, which lists all the 35 cliques and their participating member list (Table 1). “Cliques” have the strongest notion among all the other subgroup concepts. In order to strengthen the value of results, a filter of at least 3 interactions was imposed to the above clique analysis. These results clearly demonstrate that the five actors (three of whom are PD team members) have very strong relationships with every other actor in the network. A strong relationship between two actors is fostered as a result of continued interaction between them. From the above results, we can deduce that PD activity, which involved users during the design and development phases of a system, has cultivated significant level of interests within the users to make them actively participate in the sustaining and improving of the system. This stands in support of the initial hypothesis tested. USER-CENTERED EVALUATION METHODOLOGY To further assess the success of our methodology, a series of user evaluation techniques were employed. Many aspects of usability can best be studied by simply asking the users. This is especially true for issues related to the users’ subjective satisfaction and possible anxieties (Nielsen, 1993). Since the course is highly dependent on user participation, the design team has taken steps in collecting and analyzing user feedback. Evaluation of the course has been, from the beginning, an integral part of our Participatory Design implementation. First, a questionnaire was provided for collecting feedback about the general usability of the course. Secondly, a discussion board was created where users could post their questions and comments and, finally, an email address was provided through which users could contact the design team. Distributed Constructionism through Participatory Design 173 Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Cliques Members K1 (u17, u19, u21, u26, u27 , u12, u13) K2 (u17, u19, u21, u26, u27 , u12, u14) K3 (u17, u19, u21, u26, u27 , u12, u15) K4 (u17, u19, u21, u26, u27 , u7) K5 (u17, u19, u21, u26, u27 , u8) K6 (u17, u19, u21, u26, u27 , u9) K7 (u17, u19, u21, u26, u27 , u10) K8 (u17, u19, u21, u26, u27 , u11) K9 (u17, u19, u21, u26, u27 , u6) K10 (u17, u19, u21, u26, u27 , u16) K11 (u17, u19, u21, u26, u27 , u1) K12 (u17, u19, u21, u26, u27 , u18) K13 (u17, u19, u21, u26, u27 , u2) K14 (u17, u19, u21, u26, u27 , u20) K15 (u17, u19, u21, u26, u27 , u3) K16 (u17, u19, u21, u26, u27 , u22) K17 (u17, u19, u21, u26, u27 , u23, u32) K18 (u17, u19, u21, u26, u27 , u24) K19 (u17, u19, u21, u26, u27 , u25) K20 (u17, u19, u21, u26, u27 , u4) K21 (u17, u19, u21, u26, u27 , u5) K22 (u17, u19, u21, u26, u27 , u28) K23 (u17, u19, u21, u26, u27 , u29) K24 (u17, u19, u21, u26, u27 , u30, u43) K25 (u17, u19, u21, u26, u27 , u31) K26 (u17, u19, u21, u26, u27 , u33) K27 (u17, u19, u21, u26, u27 , u34) K28 (u17, u19, u21, u26, u27 , u35) K29 (u17, u19, u21, u26, u27 , u36) K30 (u17, u19, u21, u26, u27 , u37) K31 (u17, u19, u21, u26, u27 , u38) K32 (u17, u19, u21, u26, u27 , u39) K33 (u17, u19, u21, u26, u27 , u40) K34 (u17, u19, u21, u26, u27 , u41) K35 (u17, u19, u21, u26, u27 , u42) Table 1: Clique Analysis Report (Each variable under the members column represents a user and each variable under the cliques column represents a clique) 174 Zaphiris, Zacharia and Rajasekaran Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Questionnaire and Discussion Board Postings Evaluation One hundred and eighty one students, taking the online “Learn Greek Online” course in its first year of implementation, responded to an online questionnaire assessing the overall usability of the course. The questionnaire was based on the Computer System Usability Questionnaire (CSUQ) and was administered through the web-based user interface evaluation with questionnaires system provided online by Perlman (1999). CSUQ consists of 19 usability questions to which the respondent was to agree or disagree on a five-point scale, ranging from 2 (Agree) to –2 (Disagree). Table 2 shows the overall ratings for the online course for the 19 questions. A graphical representation of the results is provided in FILL IN. Next the responses to all questions were combined together and the average overall rating was plotted in FILL IN. In addition to the questionnaire, we analyzed a total of 371 postings (posted online to the discussion board from December 1998 to March 2000). These are the same postings that were analyzed using SNA in the previous sections. Question Q19 Q18 Q17 Q16 Q15 Q14 Q13 Q12 Q11 Q10 Q9 Q8 Q7 Q6 Q5 Q4 Q3 Q2 Q1 M ea n 2,0